The previous article in this series, Is Informed Consent Necessary When Artificial Intelligence is Used for Patient Care: Applying the Ethics from Justice Cardozo’s Opinion in Schloendorff v. Society of New York Hospital, argued that patients should be informed when AI is used in making their health care decisions. This article will explore the range of potential harms attributable to AI models and ask whether informed consent alone mitigates responsibility.
Every medical treatment involves some risk of harm in the pursuit of a potential benefit. Benefits may be increased function, prolonged survival, or improved quality of life. For instance, wound infections are a well known risk associated with surgery for acute appendicitis. Surgeons have a duty to inform their patients about their relative risk of a wound infection after surgery. Factors attributable to patients, such as smoking and diabetes, and factors attributable to surgeons, such as their technique and the administration of antibiotics, can affect an individual patient’s risk of developing a wound infection. Surgeons also have a duty to mitigate the patient’s risk of developing a wound infection, whether by counseling the patient to stop smoking or by administering appropriate antibiotics prior to surgery. Thus, physicians have a duty both to inform patients about their specific risk of harm and also a duty to mitigate the harm.
Patients should be informed about their risk of harm so that they can consider the information in their decision-making. A patient can weigh the risks of harm against the benefits that the patient seeks. By not informing a patient about the potential for harm, the physician fails to respect the patient’s ability to make autonomous decisions about their health care. Harkening back to Justice Cardozo’s opinion in Schloendorff v. Society of New York Hospital, an essential part of deciding what shall be done to one’s body involves considering the value of the proposed benefit and the potential for harm in attempting to realize the benefit.
While necessary, informing patients of the risk of harm is not sufficient. Physicians also have an obligation to mitigate the risk of harm to the extent possible. A physician who breaches their duty to mitigate harm to a patient bears responsibility for the harm. The extent of responsibility would be determined based on the extent of harm and the degree to which the harm could be mitigated. A physician who has the means yet fails to prevent a foreseeable and predictable harm neglects their duty to provide beneficial and non-harmful care.
How does physician responsibility and accountability for patient harm apply to AI models?
If a patient is harmed, and the AI model participated in the decision that led to harm without the patient either being informed or giving consent, the patient would have a strong claim that their autonomy was not respected. In addition, if a patient is harmed by an AI model, and the risk of harm was known and could have been mitigated but was not, the physician would have breached their duty to the patient similar to not administering antibiotics where indicated to reduce the risk of a wound infection.
What sorts of harms, traceable to the model itself, could be associated with the use of an AI for health care decisions? One of the most important harms would be intrinsic failure of the model’s accuracy in making the prediction it was programmed to make. Inaccuracy in a model’s prediction can be due to a number of factors. The model may have been trained and validated on a specific patient demographic but deployed on a different demographic. Thus, the patterns that the model has learned to associate with an outcome in the training data do not accurately predict the outcome when used to make predictions on a different set of data.
Another harm, the potential for which has been well documented in the literature, is that the model will be biased in that it will inequitably distribute benefits and burdens. This can happen because the data on which the model was trained was itself biased and the model will perpetuate that bias. An example is that in the United States, Black patients receive lower-quality comprehensive cancer care. As a result, Black patients have an overall higher cancer-related mortality than do white patients. Because of the strong statistical but not biological association between being Black and having an increased rate of cancer-related death, models trained on those data will predict that Black patients are more likely to die from cancer. The prediction is based on the patient being Black and does not take into account the structural determinants of health that influence Black patients’ access to high-quality comprehensive cancer care.
While a patient can be harmed directly by a model’s inaccurate prediction based either on unrepresentative training data or historical bias, a patient can also be harmed by an accurate prediction. One way in which this can happen is if an AI model accurately predicts a risk for harm, such as HIV or chronic kidney disease, and the predicted risk is not mitigated, and the patient goes on to develop the disease that could have been prevented. For instance, consider an AI model that very accurately predicts a patient’s risk of contracting HIV based on information in the patient’s electronic medical record (EMR). When the patient comes to their physician’s office for a visit, the physician opens the EMR and receives an alert informing the physician that the patient is predicted to be at high risk for HIV. The AI model goes on to recommend that the physician discuss the patient’s risk with them, and if the patient is determined to be at high risk for HIV, the physician should also discuss medication that can reduce their risk, HIV pre-exposure prophylaxis (PrEP).
In this example, the physician can breach their duty to mitigate harm to the patient in at least two ways. First of all, the physician can simply ignore the prediction. Or the physician can consider the prediction and reject it unilaterally without corroborating the prediction with the patient. In the first instance, the patient may have come to the physician’s office for the evaluation of knee pain following a long run and not for a comprehensive health evaluation that would involve a discussion of HIV. The physician may not have a therapeutic relationship established with the patient and be uncomfortable bringing up a sensitive issue like HIV. The physician may be skeptical of the model and resentful that they have to address HIV risk during a focused visit for another issue altogether. The physician may be running behind in the clinic and not have time to delve into a long discussion about HIV risk and prevention. Any of these considerations, and many more, may lead a physician to ignore the model’s prompt. If the physician ignores reliable information that the patient is at risk for harm, the physician has not fulfilled their duty to the patient. The physician has information that is more likely than not able to protect the patient from harm and the means to potentially mitigate the harm. It is the physician’s duty to provide patient-centered care where benefit to the patient is prioritized and harm minimized.
Besides choosing to ignore the model’s prediction, the physician could also consider it but reject it based on their clinical judgment. However, the exercise of clinical judgment does not negate the responsibility for integrating the AI prediction into a holistic assessment of the patient based on best practices. While value-laden, discussing the patient’s risk for HIV is a low-risk and potentially high-yield clinical endeavor. A physician’s clinical basis for considering and rejecting a valid and reliable prediction that a patient is at high risk for HIV is limited. This is especially true since some of the behaviors that place a patient at risk for HIV are also stigmatizing and may not be apparent in the patient’s EMR. Harm that is caused by a physician who considered but rejected an AI model’s prediction that, if acted on, could have mitigated that risk is similar to harm from a delayed diagnosis or an ineffective treatment. In all instances, harm resulted from the physician’s exercise of clinical judgment.
Finally, AI predictions that are made without either informing the patient that the prediction is being made in the first place, or without informing patients of the prediction itself, risk harming patients in ways of which they may be unaware. In the previous HIV example, if a patient is neither informed that the prediction was being made nor informed of the prediction itself, if they contract HIV, they may not even know that there was an opportunity to mitigate their risk. While HIV is a dramatic example, many other potential predictions can be made that, if the patient is not informed of the prediction, could result in harm. Most of these predictions are being made because they are believed to be beneficial to patients in some ways. However, such an approach risks normalizing the practice that patient information can be used to make whatever prediction the clinician, developer, health system, or payer desires to know.
Informing patients both that AI predictions are being made and also informing them about the prediction itself is not only an expression of respect for the patient’s autonomy in medical decision-making, but it also assures that patients, as well as clinicians, can integrate the information into their own decision-making. In addition, patients can weigh for themselves the benefits and liabilities of the potential harm as well as the potential for mitigation. AI predictions are not like lab tests and radiology studies and they do not require the same kind of clinical consideration possessed by physicians. Unlike other kinds of medical data, patients can interpret most AI predictions for themselves and decide whether and how to mitigate their risk of harm. The 21st Century Cures Act requires that some information used to make clinical decisions be shared with patients. This requirement should apply to predictions made by AI models.
